<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"  
  "http://www.w3.org/TR/html4/loose.dtd">  
<html > 
<head><title>Chinese Word Segmentation</title> 
<meta http-equiv="Content-Type" content="text/html; charset=gbk"> 
<meta name="generator" content="TeX4ht (http://www.cse.ohio-state.edu/~gurari/TeX4ht/)"> 
<meta name="originator" content="TeX4ht (http://www.cse.ohio-state.edu/~gurari/TeX4ht/)"> 
<!-- html --> 
<meta name="src" content="segmentor.tex"> 
<meta name="date" content="2013-03-28 00:52:00"> 
<link rel="stylesheet" type="text/css" href="segmentor.css"> 
</head><body 
>
   <div class="maketitle">
                                                                          

                                                                          
                                                                          

                                                                          

<h2 class="titleHead">Chinese Word Segmentation</h2>
<div class="author" ></div><br />
<div class="date" ><span 
class="ptmr7t-x-x-144">March 28, 2013</span></div>
   </div>
   <h3 class="sectionHead"><span class="titlemark">1    </span> <a 
 id="x1-10001"></a>How to compile</h3>
<!--l. 18--><p class="noindent" >Suppose that ZPar has been downloaded to the directory <span 
class="ptmri7t-x-x-120">zpar</span>. To make the segmentor
system, type <span 
class="ptmri7t-x-x-120">make segmentor</span>. This will create a directory <span 
class="ptmri7t-x-x-120">zpar/dist/segmentor</span>, in which
there are two files: <span 
class="ptmri7t-x-x-120">train </span>and <span 
class="ptmri7t-x-x-120">segmentor</span>. The file <span 
class="ptmri7t-x-x-120">train </span>is used to train a segmentation
model,and the file <span 
class="ptmri7t-x-x-120">segmentor </span>is used to segment new texts using a trained segmentation
model.
   <h3 class="sectionHead"><span class="titlemark">2    </span> <a 
 id="x1-20002"></a>Format of inputs and outputs</h3>
<!--l. 20--><p class="noindent" >The input files to the segmentor are formatted as a sequence of Chinese characters. An
example input is: <br 
class="newline" /><br 
class="newline" />&#x00A0;&#x00A0;&#x00A0;&#x00A0;&#x00A0;                             ZPar<span 
class="gbksong47-x-x-120">&#x53ef;</span><span 
class="gbksong61-x-x-120">&#x4ee5;</span><span 
class="gbksong41-x-x-120">&#x5206;</span><span 
class="gbksong58-x-x-120">&#x6790;</span><span 
class="gbksong64-x-x-120">&#x4e2d;</span><span 
class="gbksong58-x-x-120">&#x6587;</span><span 
class="gbksong43-x-x-120">&#x548c;</span><span 
class="gbksong62-x-x-120">&#x82f1;</span><span 
class="gbksong58-x-x-120">&#x6587; </span><br 
class="newline" /><br 
class="newline" />The output files contain space-separated words: <br 
class="newline" /><br 
class="newline" />&#x00A0;&#x00A0;&#x00A0;&#x00A0;&#x00A0;                             ZPar <span 
class="gbksong47-x-x-120">&#x53ef;</span><span 
class="gbksong61-x-x-120">&#x4ee5; </span><span 
class="gbksong41-x-x-120">&#x5206;</span><span 
class="gbksong58-x-x-120">&#x6790; </span><span 
class="gbksong64-x-x-120">&#x4e2d;</span><span 
class="gbksong58-x-x-120">&#x6587; </span><span 
class="gbksong43-x-x-120">&#x548c; </span><span 
class="gbksong62-x-x-120">&#x82f1;</span><span 
class="gbksong58-x-x-120">&#x6587; </span><br 
class="newline" /><br 
class="newline" />The output format is also the format of training files for the <span 
class="ptmri7t-x-x-120">train </span>executable.
<br 
class="newline" />Both input and output files must be encoded in <span 
class="ptmri7t-x-x-120">utf8</span>. Here is a <a 
href="seg_files/gb2utf.py" >script</a> that transfers files
that are encoded in <span 
class="ptmri7t-x-x-120">gb </span>to the <span 
class="ptmri7t-x-x-120">utf8 </span>encoding.
   <h3 class="sectionHead"><span class="titlemark">3    </span> <a 
 id="x1-30003"></a>How to train a model</h3>
<!--l. 36--><p class="noindent" >To train a model, use <br 
class="newline" /><br 
class="newline" />&#x00A0;&#x00A0;&#x00A0;&#x00A0;&#x00A0;                             zpar/dist/segmentor/train <span 
class="cmmi-12">&#x003C;</span>train-file<span 
class="cmmi-12">&#x003E; &#x003C;</span>model-file<span 
class="cmmi-12">&#x003E; &#x003C;</span>number
of iterations<span 
class="cmmi-12">&#x003E; </span><br 
class="newline" /><br 
class="newline" />For example, using the <a 
href="seg_files/train.txt" >example train file</a>, you can train a model by <br 
class="newline" /><br 
class="newline" />&#x00A0;&#x00A0;&#x00A0;&#x00A0;&#x00A0;                             zpar/dist/segmentor/train train.txt model 1 <br 
class="newline" /><br 
class="newline" />After training is completed, a new file <span 
class="ptmri7t-x-x-120">model </span>will be created in the current
directory, which can be used to segment raw Chinese sentences. The above
command performs training with one iteration (see Section&#x00A0;<a 
href="#x1-60006">6<!--tex4ht:ref: tuning --></a>) using the training
file.
   <h3 class="sectionHead"><span class="titlemark">4    </span> <a 
 id="x1-40004"></a>How to segment new texts</h3>
<!--l. 50--><p class="noindent" >To apply an existing model to segment new texts, use <br 
class="newline" /><br 
class="newline" />&#x00A0;&#x00A0;&#x00A0;&#x00A0;&#x00A0;                             zpar/dist/segmentor/segmentor <span 
class="cmmi-12">&#x003C;</span>model<span 
class="cmmi-12">&#x003E; </span><span 
class="cmr-12">[</span><span 
class="cmmi-12">&#x003C;</span>input-file<span 
class="cmmi-12">&#x003E;</span><span 
class="cmr-12">]</span>
<span 
class="cmr-12">[</span><span 
class="cmmi-12">&#x003C;</span>output-file<span 
class="cmmi-12">&#x003E;</span><span 
class="cmr-12">] </span><br 
class="newline" /><br 
class="newline" />where the input file and output file are optional. If the output file is not specified,
segmented texts will be printed to the console. If the input file is not specified, raw texts
will be read from the console. For example, using the model we just trained, we can
segment <a 
href="seg_files/input.txt" >an example input</a> by <br 
class="newline" /><br 
class="newline" />&#x00A0;&#x00A0;&#x00A0;&#x00A0;&#x00A0;                             zpar/dist/segmentor/segmentor model input.txt output.txt
<br 
class="newline" /><br 
class="newline" />The output file contains automatically segmented texts.
   <h3 class="sectionHead"><span class="titlemark">5    </span> <a 
 id="x1-50005"></a>Outputs and evaluation</h3>
<!--l. 64--><p class="noindent" >Automatically segmented texts contain errors. In order to evaluate the quality of the
outputs, we can manually specify the segmentation of a sample, and compare the
outputs with the correct sample. <br 
class="newline" />A manually specified segmentation of the input file is given in <a 
href="seg_files/reference.txt" >this example reference
file</a>. Here is a <a 
href="seg_files/evaluate.py" >Python script</a> that performs automatic evaluation. <br 
class="newline" />Using the above <span 
class="ptmri7t-x-x-120">output.txt </span>and <span 
class="ptmri7t-x-x-120">reference.txt</span>, we can evaluate the accuracies by typing
<br 
class="newline" /><br 
class="newline" />&#x00A0;&#x00A0;&#x00A0;&#x00A0;&#x00A0;                             python evaluate.py output.txt reference.txt <br 
class="newline" /><br 
class="newline" />You can find the precision, recall, and f-score here.
   <h3 class="sectionHead"><span class="titlemark">6    </span> <a 
 id="x1-60006"></a>How to tune the performance of a system</h3>
<!--l. 77--><p class="noindent" >The performance of the system after one training iteration may not be optimal. You can
try training a model for another few iterations, after each you compare the performance.
You can choose the model that gives the highest f-score on your test data. We
conventionally call this test file the development test data, because you develop a
segmentation model using this. Here is <a 
href="seg_files/test.sh" >a shell script</a> that automatically trains
the segmentor for 30 iterations, and after the <span 
class="cmmi-12">i</span>th iteration, stores the model
file to model.<span 
class="cmmi-12">i</span>. You can compare the f-score of all 30 iterations and choose
model.<span 
class="cmmi-12">k</span>, which gives the best f-score, as the final model. In this file, there is a
variable called <span 
class="ptmri7t-x-x-120">segmentor</span>. You need to set this variable to the relative directory of
<span 
class="ptmri7t-x-x-120">zpar/dist/segmentor</span>.
<!--l. 79--><p class="noindent" >
   <h3 class="sectionHead"><span class="titlemark">7    </span> <a 
 id="x1-70007"></a>Source code</h3>
<!--l. 80--><p class="noindent" >The source code for the segmentor can be found at <br 
class="newline" /><br 
class="newline" />&#x00A0;&#x00A0;&#x00A0;&#x00A0;&#x00A0;                             zpar/src/chinese/segmentor/implementation/SEGMENTOR_IMPL
                                                                          

                                                                          
<br 
class="newline" /><br 
class="newline" />where SEGMENTOR_IMPL is a macro defined in <span 
class="ptmri7t-x-x-120">Makefile</span>, and specifies a specific
implementation for the segmentor.
   <h3 class="likesectionHead"><a 
 id="x1-80007"></a>References</h3>
<!--l. 87--><p class="noindent" >
     <div class="thebibliography">
     <p class="bibitem" ><span class="biblabel">
  [1]<span class="bibsp">&#x00A0;&#x00A0;&#x00A0;</span></span><a 
 id="Xbib-1"></a>Yue  Zhang  and  Stephen  Clark.  2007.  Chinese  Segmentation  Using  a
     Word-based Perceptron Algorithm. In <span 
class="ptmri7t-x-x-120">Proc. of ACL</span>. pages 840-847.</p></div>
    
</body></html> 

                                                                          


